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ABSTRACT 



This paper uses the data from a longitudinal study to assess 
the impact of technology introduction in liberal arts education at a 
non-technical small private university in the United States. The data was 
collected over four academic years, and this "post hoc" analysis uses the 
data collected in the spring of 1999 from a sample of students who responded 
to a questionnaire that was mailed to them. The questionnaire included nearly 
50 attitude items measured on a five-point Likert-type scale. Of these 50, 
there were several items that were specific to the local conditions of the 
university. The remaining items are analyzed in this paper. Analysis included 
two components: first, a factor analysis that elicited 4 factors on the basis 
of 25 items that were selected as the general attitude items in the 
questionnaire, and second, the items that loaded on the same factor were 
subject to reliability tests to explore the levels of reliability of the 
similar items. The 25 items were analyzed using a varimax rotation and this 
resulted in four major factor loadings. Four tables report the factor 
loadings, Alpha value for each factor. Alpha values after dropping item with 
the least weight, and attitude items. The results suggest that there are 
specific clusters of items that can be used to measure the effectiveness by 
using the surrogate measure of attitudes towards the use of computers in 
teaching. (Contains 17 references.) (AEF) 
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Abstract: This paper uses the data from a longitudinal study to conduct a series of 
factor analysis to test the reliability of a set of items to measure the effectiveness of the 
use of computers in higher education. The results suggest that there are specific 
clusters of items that can be used to measure the effectiveness by using the surrogate 
measure of attitudes towards the use of computers in teaching. 



Introduction 



One of the thorniest problems following the introduction of computers in education has been 
assessing the efficacy of technology in improving the quality of education. There is little doubt that 
technology and education are now intimately associated and there is a growing application of technological 
tools in the educational process (see, e.g., Brown, 2000; Oblinger and Rush, 1997). This increased 
application of technology is constantly calling for the need for better assessment of the effects of 
technology and there are relatively few good methods of evaluating this efficacy (Reeves, 1998; Woodrow, 
1994 ; Woodrow, 1996). There are several problems related to the process of evaluation with the primary 
being the lack of a good measurement tool to track changes in the educational outcome. This problem is 
also related to the fact that "learning" itself is a slippery term and defies universal definition, making the 
quantification and measurement of learning a tricky process. Indeed, without a robust definition it becomes 
particularly difficult to measure it. 

It has been suggested that learning can be measured using other "surrogate" measuring techniques 
such as the evaluation of the changes in the attitudes towards learning and the changes in the learning 
process . Based on the examination of a series of assessment projects, Mitra and Hullett (1997) has 
suggested that attitudes towards technology can perform as an alternative way to measure the effectiveness 
of the introduction of technology in the learning place. This approach has been applied to various 
academic scenarios to report on the ways in which technology continues to impact the process of teaching 
and learning (see, e.g., Lawson and Pelzer, 1999; McQueen and Fleck, 1999; Merisotis and Phipps, 1999; 
Mitra, et. al., in press; Mitra, et. al., 2000; Mitra, et. al., 1999; Mitra and Hazen, 1999; Mitra, 1998; Phipps 
and Merisotis, 1999). However, what has still remained a challenge is trying to find a particularly robust 
set of measures that would address all the different aspects of the effects of technology in teaching. 
Woodrow (1991) reported on the analysis of four different attitude scales and all of them tended to measure 
different attributes of the outcome of the application of technology in teaching. Furthermore, with the 
changes in technology some of the specific aspects of the measurement continue to change as well, thus 
making some of the scales inappropriate for the current mode of technology intervention. 

In this paper it is suggested that it is possible to identify some specific aspects of the outcome of 
the technology use in teaching where the factors being measured are not necessarily technologically 
determined. In stead, the factors being measured are more fundamental constructs of the human- 
technology interaction in the educational setting and thus can be applied more broadly to several different 
assessment situations and can be more universally applied. Using the data from an ongoing longitudinal 
study of the application of technology in an American private liberal arts institution, it is possible to show 
that there are certain fundamental aspects of the effects of the application of technology that can be 
measured using reliable attitude items. 
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This paper utilizes the data from a longitudinal study to assess the impact of technology 
introduction in liberal arts education at a non-technical small private university in the United States. The 
data was collected over four academic years and this post hoc analysis uses the data collected in the spring 
of 1999 from a sample of students who responded to a questionnaire mailed to them. The questionnaire 
was developed by conducting a series of focus group meetings that explored the various technology-related 
issues that the students were facing with the universal and standardized introduction of technology in 
teaching. The questionnaire included nearly fifty attitude items measured on a five-point Lickert-type 
scale. Of these fifty, there were several items that were specific to the local conditions of the university. 
The remaining items have been analyzed here. The entire questionnaire also consisted of usage questions 
and demographic questions. 

Analysis and Results 

The analysis included two components: first, a factor analysis that elicited four factors on the basis 
of twenty-five items that were selected as the general attitude items in the^ questionnaire and second, the 
items that loaded on the same factor were subject to reliability tests (Cronbach's alpha) to explore the levels 
of reliability of the similar items. 

The 25 items were analyzed using a varimax rotation and this resulted in four major factor 
loadings. These are reported in the next table. 



Table 1: Factor loadings 



Factor 1 


Loading 


1. Talking face-to-face with professors is generally gratifying 


.984 


2. Computers enable me to interact more with professors 


.488 


3. The use of computers positively impacts the social life 


.976 


4. The use of computers makes the academic climate of Wake Forest intellectually 
exciting 


.984 


5. The use of computers is increasing cooperative learning at WFU 


.749 


6. Computer mediated communication makes it easy to maintain relationships 


.982 


7. Computers provide a non-threatening way to communicate 


.628 


8. Computers allow me to communicate with people I would not normally be able to 
communicate with 


.921 


9. When communicating using a computer one does not need to be formal 


.976 


10. The use of computers is strengthening the academic climate 


.983 


Factor 2 




1 1. Computers are effective for communicating with other students about class related 
work 


.824 


12. Computers are effective for communicating with faculty about class related work 


.888 


13. Computers are effective for communicating with other students about non-course 
related subjects 


.802 


14. Computers are effective for communicating with faculty about non-course related 
topics 


.691 


15. Communicating with professors by e-mail is generally gratifying 


.628 


Factor 3 




16. I would stay away from classes that do not use computers 


.920 


17. The computerization process poses a threat to my privacy 


.810 


1 8. The adoption of computers will give more power to the instructors 


.922 


19. Computer use increases the usual college work load 


.293 


Factor 4 




20. There is a need for classes which teach "how to" use a computer 


.142 


21. I prefer classes in which I get to use computers 


.502 


22. Computer use helps me better understand the course material 


.571 


23. I prefer classes with "hands-on" computer experiences 


.610 
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24. I expect that I will be required to use the computer 


.616 


25. I expect ALL instructors at Wake Forest will use the computer in teaching 


.687 



The factor analysis was followed by the computation of the inter-item reliability for the items within each 
of the factors. This correlation was high, with the Chronbach’s alphas for the four factors reported in the 
next table. 



Table 2: Alpha value for each factor 



Factor Number 


Alpha 


1 


.93 


2 


.81 


3 


.75 


4 


.36 



While the alpha values reported in Table 2 are relatively high, a second set of alpha values were computed 
after dropping from the set the item with the lowest factor weight from each of the factors reported in Table 
1 . This resulted in higher alpha values and thus a more robust set of items. These adjusted alpha values are 
reported in the next table. 



Table 3: Alpha values after dropping item with least weight 



Factor Number 


Alpha 


1 (drop Item 2) 


.96 


2 (drop Item 15) 


.81 


3 (drop Item 19) 


.81 


4 (drop Item 20) 


.51 



These results suggest that it is possible to propose a computer attitude scale with 21 items which could 
serve as a surrogate measure for learning outcomes by providing an evaluation of the attitudes of students 
towards the use of computers in teaching. The items all could use a five-point Lickert type scale with '5* 
being 'strongly agree,’ '4' being ’agree,’ ’3’ being ’no opinion,’ ’2' being ’disagree,’ and T being strongly 
disagree.' The items are now presented with slight alterations in the reference to a specific university so 
that the items can be used at any place of higher education. 



Table 4: Attitude items 



Talking face-to-face with professors is generally gratifying 


The use of computers positively impacts the social life 


The use of computers makes the academic climate of the university intellectually exciting 


The use of computers is increasing cooperative learning at the university 


Computer mediated communication makes it easy to maintain relationships 


Computers provide a non-threatening way to communicate 


Computers allow me to communicate with people I would not normally be able to communicate with 


When communicating using a computer one does not need to be formal 


The use of computers is strengthening the academic climate 


Computers are effective for communicating with other students about class related work 


Computers are effective for communicating with faculty about class related work 


Computers are effective for communicating with other students about non-course related subjects 


Computers are effective for communicating with faculty about non-course related topics 


I would stay away from classes that do not use computers 


The computerization process poses a threat to my privacy 


The adoption of computers will give more power to the instructors 


I prefer classes in which I get to use computers 


Computer use helps me better understand the course material 


I prefer classes with ’’hands-on” computer experiences 
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I expect that I will be required to use the computer 

I expect ALL instructors will use the computer in teaching 



Discussion 

The analysis presented here first suggests that it is possible to develop standardized items that can 
be used to measure the attitude of students towards the introduction of computers in teaching. The items, 
by clustering around four factors, also represent specific attributes of the effects of computerization that can 
become yardsticks for the more urgent question of measuring the effectiveness of computers in enhancing 
learning. In presenting the findings from several different studies, Mitra and Hullett (1997) have indicated 
that "predsipositions" play an important role in the perception of the effectiveness of computer aided 
instruction. Within the notion of predisposition, the authors included issues such as comfort with 
computers, the rapidity of the introduction of computers and other such perceptual ideas. Additionally, the 
authors noted that technographics, context and technological sophistication could also play roles in the 
determination of effectiveness of computers in instruction. 

The results of this study help to support the findings of Mitra and Hullet (1997), but also suggest 
that the idea of measuring predsipositions need to be made more elaborate and must include a large array of 
perceptual parameters. Indeed, some of these parameters are no longer only predsipositions but also 
include perceptions that develop as computer aided instruction becomes more prevalent on college 
campuses. In other words, the perceptions that can determine the effectiveness are not only the ones that 
the students possess as they are initiated to computer aided instruction, but the effectiveness could also 
depend on perceptions that develop with continued exposure to computer aided instruction. Indeed, the 
four factors identified here represent four major perceptual categories all of which arguably can have an 
impact on the perceived effectiveness of the computerization process. This is supported also by arguments 
made by authors such as Pacey (1983) and Negroponte (1995) who have suggested that the effectiveness of 
computers often depends on the way in which the users might perceive the values associated with the 
"practice" of using computers. Thus, by measuring the values and expectations related to computers in 
teaching, it might be possible to arrive at a more reliable measure of the effectiveness. The data from this 
study suggests that there are four key values that can be measured as represented in the four factors 
identified in the analysis. 

The first parameter that needs to be measured can be called "climate of interaction." Clearly, the 
computer has become a key form of an interaction device for the students and the effectiveness of the 
computer as an interactive tool can have a large impact on the way in which the computer becomes 
effective in instruction. Consequently, by knowing how well the environment supports the interactive 
applications of the computer could have a lot to do with how students evaluate the effectiveness of the 
computer in education. Extrapolating from there, it is also possible to argue that if indeed the computer 
enhances interaction then it could begin to play a critical role in transforming the instructional environment. 
Thus it is important to measure the way in which students perceive the enhancements in interaction with the 
introduction of computers. Interestingly, this enhancement is not only with the use of computer mediated 
communication, but the very presence of the computer and the accompanying shifts in the climate of the 
institution can also affect the more traditional forms of communication such as "face-to-face" interactions. 
The data suggests that such attributes should also be measured to get a sense of the changes in the climate 
of interaction that follows the introduction of technology. Thus the following items could be used to 
measure the climate of interaction. 

□ Talking face-to-face with professors is generally gratifying 

□ The use of computers positively impacts the social life 

□ The use of computers makes the academic climate of the university intellectually exciting 

□ The use of computers is increasing cooperative learning at the university 

□ Computer mediated communication makes it easy to maintain relationships 

□ Computers provide a non-threatening way to communicate 

□ Computers allow me to communicate with people I would not normally be able to communicate with 

□ When communicating using a computer one does not need to be formal 

□ The use of computers is strengthening the academic climate 



5 



Page 1330 



The second parameter that emerges as important is also related to communication and can be 
labeled, "communicative effectiveness." Unlike the earlier item which is broad-based and addresses issues 
related to the overall climate of an academic institution, this second factor includes attributes that focus 
primarily on the role of the computer in the pragmatic act of communicating. It can be argued that a 
change in the perception of the effectiveness of the computer in facilitating communication could have an 
impact on the learning outcome. Consequently, it is important to be able to measure how effective the 
computer is considered to be in communicating with various groups with whom students typically have to 
communicate in the course of their academic activities in college. The following items refer to the 
communicative effectiveness of the computer in education. 

□ Computers are effective for communicating with other students about class related work 

□ Computers are effective for communicating with faculty about class related work 

□ Computers are effective for communicating with other students about non-course related subjects 

□ Computers are effective for communicating with faculty about non-course related topics 

The third parameter that is important to measure to obtain an evaluation of the computer in 
instruction can be called the "abstract expectations" of the students as they are placed in a computer aided 
instruction situation. It is clear that the promises of technology tend to alter what students can expect from 
the computer and the instructional situation that they are placed in. These expectations, and how well they 
are met, can have a direct impact on the way in which students react to the overall use of technology in 
instruction. There are many facets of these abstract expectation and they include expectations about the 
level of use of computers in learning; apprehensions about the loss of privacy and concerns about the 
increases in the power inequity between the teacher and the student. In many cases these might strictly be 
perceptions and the changes are not as dramatic as the students might think they are, but it is still important 
to measure these expectations because people with different expectations could easily have different 
assessments of the effectiveness of the computer in teaching (see, e.g., Mitra, et. al., 1999). Therefore, the 
following items can be used to measure the abstract expectations related to the introduction of computers in 
teaching. 

□ I would stay away from classes that do not use computers 

□ The computerization process poses a threat to my privacy 

□ The adoption of computers will give more power to the instructors 

□ I prefer classes in which I get to use computers 

Following from the notion of abstract expectations, there is a fourth parameter that should also be 
measured to develop an assessment of the effectiveness of the computer in teaching. The fourth parameter 
can be called, "pragmatic expectations." These items also deal with the notion of the benefits and burdens 
of technology, but focus more on the specific ways in which student's anticipate using the computer in the 
process of learning. Students entering a computer-enriched environment often come with specific 
expectations about what the computer could do to transform the learning process. If they perceive that 
there will be a positive transformation it is possible that their learning will be enhanced as well. Thus, it is 
important to measure what the students expect in terms of what the computer can do to transform their 
comprehension of the material being taught. This change is in turn related to the ways in which the 
students feel about the extent of their computer use with "hands on" application as well as the breadth of 
the computer use in terms of the number of classes in which they might be asked to use the computer. As 
pointed out in earlier research, these issues can impact the students’ general assessment of the role and 
effectiveness of the computer in the practice of teaching. These items take the focus of attention away from 
a technologically determined, technology-centric view to a more sociologically determined focus on the 
practice of technology (Pacey, 1983). The following items attempt to measure this aspect of the pragmatic 
exceptions arising from the practice of using computers in teaching. 

□ Computer use helps me better understand the course material 

□ I prefer classes with "hands-on" computer experiences 

□ I expect that I will be required to use the computer 

□ I expect ALL instructors will use the computer in teaching 
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In summary, this paper and the data offers a valid and reliable measurement of four key 
parameters that can serve as indicators for the assessment of the effectiveness of computers in teaching. 
While these measurements offer an initial glimpse on the thorny question of measuring the effectiveness of 
computer assisted teaching, they do not, however, claim to be direct measurements of the teaching 
effectiveness. These items only offer alternative and surrogate measurements which, however, can be 
coupled with more direct measurements. 
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